Friday, February 3rd 2017, had been a good day – at first: PHPUnit 6.0.0 had just been released, containing many new features. What had fallen by the wayside was the support for PHP 5. It seemed as if all signs were thoroughly pointing to the future.

But then something strange happened.

Millions of tests suddenly cried out

Within a very short period of time lots of complaints about failing tests were popping up on Twitter. Symfony-applications stopped running, Travis was waving the red flag. What had happened?

Everyone who was watching the signals on social media was wondering about the events unfolding. Because in addition to disappointed users, there were many people celebrating the release and its improvements in performance

PHPUnit 6 – the interview

In order to get to the bottom of things, we talked to Sebastian Bergmann about these events and the current situation. As it turns out, the problems were more complex than expected. Obviously, official announcements made long before the release are sometimes not enough to prevent confusion. But read for yourself.

Sebastian, following the release of PHPUnit 6 complaints about problems with tests quickly popped up on Twitter. Several things seemed to have happened on different levels. What were those levels, what had happened exactly?

Sebastian Bergmann: Phew… A lot of stuff had happened all at once. PHPUnit 6.0 only supports PHP 7.0 and PHP 7.1, meaning it does not work with PHP 5 anymore. If you try to run PHPUnit 6.0 with PHP 5, what you get is, of course, an error.

The new major version implements one of the PHP community’s long-held wishes, namely that PHPUnit’s code uses namespaces. PHPUnit_Framework_TestCase does not exist anymore, test classes now need to inherit from PHPUnit\Framework\TestCase.

In addition, two default configurations have changed in PHPUnit 6.0. This invoked problems in some test suites out there. On the one hand, PHPUnit is now per default complaining about tests which do not test anything, i.e. test which do not call an assertion, which do not expect an exception with expectException() or @expectedException or configure expectations on a mock object. Test which are in my opinion worthless and therefore risky, and which on the other hand had not been criticized by PHPUnit 5.7 per default, are now treated differently per default.

Furthermore, PHPUnit does not create a backup of all global and super-global variables before every tests anymore – and it does not recreate these backups after every tests anymore. Tests of code which manipulate global or super-global variables can now fail in the default configuration.

As far as PHPUnit’s default configuration is concerned, I believe it is about time that developers who write modern, clean, object-oriented code, code which is not dependent on the global system status, that these developers no longer need to pay the price for the fact that old code is made testable through the mechanism of Global State Backups in the default configuration. The main reason being that the creating and restoring of global and super-global variables takes up memory and time.

In the end, the release of PHPUnit 6.0 happened during a week in which the development of PHP 7.2, wich is expected to be released in December of 2017, has clearly gained steam.

The „problem child“ Symfony in combination with PHPUnit is a special case. What has been done in this respect?

Sebastian: The Symfony developers would like the current version – Symfony 3 – to be compatible with PHP 5.5. Therefore, you need to be able to (also) run Symfony’s test suite with PHP 5.5. PHPUnit 4.8 was the last version which supported PHP 5.5. In the meantime, PHP 5.5 has reached its „end of life“ on July 21 2016 while PHPUnit has done the same on February 3 2017.

In order to help the Symfony developers I published one final release of PHPUnit 4.8 two weeks ago. This release now also contains the Forward Compatibility Layer which PHPUnit 5 has had for quite a while. Thanks to that, PHPUnit 4.8 now also knows how to handle PHPUnit\Framework\TestCase instead of PHPUnit_Framework_TestCase. This helps the Symfony developers running their tests with PHPUnit 4.8, PHPUnit 5.7, and PHPUnit 6.0.

Another problem which came up was that PHPUnit 6 was not compatible to PHP 7.0.0 up until 7.0.12. What exactly had happened there?

Sebastian: *sigh* This was a bug which irritated me more than any other in the recent past. For the new version, I’ve moved PHPUnit’s classes with PHPstorm in a namespace (each) and renamed them afterwards. This worked fine and without Phpstorm’s automated refactorings I would have gone crazy. But there were two exceptions. Since I had only tested with current, stable PHP-versions – PHP 7.0.15 and PHP 7.1.1 to be exact – both locally and on Travis, I didn’t notice the incorrect renamings. Therefore, I could’t either replicate nor understand some of the bug reports coming in after the release.

Only after I had configured Travis in a way that PHPUnit’s test suite ran properly with all releases of PHP (7.0.0, 7.0.1, …, 7.1.0, 7.1.1), I was able to see that PHPUnit 6.0 was not compatible with PHP 7.0.0 up to 7.0.12. One chat with core developer Nikita Popov later I knew what had changed in PHP 7.0.13. Since PHP 7.0.13 the errors that the automated refactoring had caused in respect to namespaces are no longer errors. The reason being that PHP now implements the so called namespace shadowing correctly. Notwithstanding, I corrected the errors in PHPUnit 6.0, so that the new version can be used with older versions than 7.0.13.

With PHPUnit you are now striking a path which seems to be tough, to say the least: the support for PHP 5 is end-of-life, so you’re concentrating only on the supported versions. Is reality ready for this? Meaning: is PHP 7 proliferated enough in daily use?

Sebastian: PHPUnit is expected to be working with all current versions of PHP. That’s okay. Something I neither deem sensible nor achievable from my side is the compatibility of current PHPUnit versions with outdated PHP-versions. Developers using PHP 5.6 (or even older versions) face a different set of problems all together in my opinion. And the current version of PHPUnit not working with their tests is the least of them. I discussed this problem extensively in my article „PHP 5: Active Support Ends. Now what?“ when the active support of PHP 5.6 had ended.

As far as statistics, like those by Jordi (PHP Version Stats – 2016.Edition) or surveys like those done by entwickler.de are concerned: My interpretation is that at least new projects are being implemented with PHP 7. This corresponds to my everyday experiences as a consultant.

As far as exisiting projects using old PHPUnit-versions are concerned: only because there is a new version of PHPUnit does not mean that the old one stops working. You just don’t get new features. And if you find a bug, you can keep it.

If we approach the topic from the perspective of a developer who takes pride in his code’s quality: Did the outcry in the community alarm you to some extent? Because after all it seems like a tool like PHPUnit has become part of a PHP developer’s standard repertoire.

Sebastian: Of course, I’m not worried by certain concrete problems of PHPUnit users. But on a more abstract level, I think you’re right. Yes, it is alarming that so many developers are depending on PHPUnit in their everyday projects
– that it causes a stir if there are problems with PHPUnit.